feat: add `stats/incr/wstdev` #4914

anandkaranubc · 2025-01-26T18:36:10Z

Resolves #47

Description

What is the purpose of this pull request?

This pull request:

Adds the JS implementation of the incremental weighted standard deviation.

Related Issues

Does this pull request have any related issues?

This pull request:

resolves #47

Questions

Any questions for reviewers of this pull request?

No.

Other

Any other information relevant to this pull request? This may include screenshots, references, and/or implementation notes.

This PR will also serve as a reference implementation for [RFC]: add stats/incr/wvariance #46

Checklist

Please ensure the following tasks are completed before submitting this pull request.

Read, understood, and followed the contributing guidelines.

@stdlib-js/reviewers

--- type: pre_commit_static_analysis_report description: Results of running static analysis checks when committing changes. report: - task: lint_filenames status: passed - task: lint_editorconfig status: passed - task: lint_markdown status: na - task: lint_package_json status: na - task: lint_repl_help status: na - task: lint_javascript_src status: passed - task: lint_javascript_cli status: na - task: lint_javascript_examples status: na - task: lint_javascript_tests status: na - task: lint_javascript_benchmarks status: na - task: lint_python status: na - task: lint_r status: na - task: lint_c_src status: na - task: lint_c_examples status: na - task: lint_c_benchmarks status: na - task: lint_c_tests_fixtures status: na - task: lint_shell status: na - task: lint_typescript_declarations status: na - task: lint_typescript_tests status: na - task: lint_license_headers status: passed ---

--- type: pre_commit_static_analysis_report description: Results of running static analysis checks when committing changes. report: - task: lint_filenames status: passed - task: lint_editorconfig status: passed - task: lint_markdown status: na - task: lint_package_json status: na - task: lint_repl_help status: na - task: lint_javascript_src status: na - task: lint_javascript_cli status: na - task: lint_javascript_examples status: passed - task: lint_javascript_tests status: passed - task: lint_javascript_benchmarks status: passed - task: lint_python status: na - task: lint_r status: na - task: lint_c_src status: na - task: lint_c_examples status: na - task: lint_c_benchmarks status: na - task: lint_c_tests_fixtures status: na - task: lint_shell status: na - task: lint_typescript_declarations status: na - task: lint_typescript_tests status: na - task: lint_license_headers status: passed ---

--- type: pre_commit_static_analysis_report description: Results of running static analysis checks when committing changes. report: - task: lint_filenames status: passed - task: lint_editorconfig status: passed - task: lint_markdown status: na - task: lint_package_json status: na - task: lint_repl_help status: na - task: lint_javascript_src status: na - task: lint_javascript_cli status: na - task: lint_javascript_examples status: na - task: lint_javascript_tests status: na - task: lint_javascript_benchmarks status: na - task: lint_python status: na - task: lint_r status: na - task: lint_c_src status: na - task: lint_c_examples status: na - task: lint_c_benchmarks status: na - task: lint_c_tests_fixtures status: na - task: lint_shell status: na - task: lint_typescript_declarations status: passed - task: lint_typescript_tests status: passed - task: lint_license_headers status: passed ---

--- type: pre_commit_static_analysis_report description: Results of running static analysis checks when committing changes. report: - task: lint_filenames status: passed - task: lint_editorconfig status: passed - task: lint_markdown status: na - task: lint_package_json status: na - task: lint_repl_help status: passed - task: lint_javascript_src status: na - task: lint_javascript_cli status: na - task: lint_javascript_examples status: na - task: lint_javascript_tests status: na - task: lint_javascript_benchmarks status: na - task: lint_python status: na - task: lint_r status: na - task: lint_c_src status: na - task: lint_c_examples status: na - task: lint_c_benchmarks status: na - task: lint_c_tests_fixtures status: na - task: lint_shell status: na - task: lint_typescript_declarations status: na - task: lint_typescript_tests status: na - task: lint_license_headers status: passed ---

--- type: pre_commit_static_analysis_report description: Results of running static analysis checks when committing changes. report: - task: lint_filenames status: passed - task: lint_editorconfig status: passed - task: lint_markdown status: na - task: lint_package_json status: passed - task: lint_repl_help status: na - task: lint_javascript_src status: na - task: lint_javascript_cli status: na - task: lint_javascript_examples status: na - task: lint_javascript_tests status: na - task: lint_javascript_benchmarks status: na - task: lint_python status: na - task: lint_r status: na - task: lint_c_src status: na - task: lint_c_examples status: na - task: lint_c_benchmarks status: na - task: lint_c_tests_fixtures status: na - task: lint_shell status: na - task: lint_typescript_declarations status: na - task: lint_typescript_tests status: na - task: lint_license_headers status: passed ---

--- type: pre_commit_static_analysis_report description: Results of running static analysis checks when committing changes. report: - task: lint_filenames status: passed - task: lint_editorconfig status: passed - task: lint_markdown status: passed - task: lint_package_json status: na - task: lint_repl_help status: na - task: lint_javascript_src status: na - task: lint_javascript_cli status: na - task: lint_javascript_examples status: na - task: lint_javascript_tests status: na - task: lint_javascript_benchmarks status: na - task: lint_python status: na - task: lint_r status: na - task: lint_c_src status: na - task: lint_c_examples status: na - task: lint_c_benchmarks status: na - task: lint_c_tests_fixtures status: na - task: lint_shell status: na - task: lint_typescript_declarations status: na - task: lint_typescript_tests status: na - task: lint_license_headers status: passed --- --- type: pre_push_report description: Results of running various checks prior to pushing changes. report: - task: run_javascript_examples status: na - task: run_c_examples status: na - task: run_cpp_examples status: na - task: run_javascript_readme_examples status: na - task: run_c_benchmarks status: na - task: run_cpp_benchmarks status: na - task: run_fortran_benchmarks status: na - task: run_javascript_benchmarks status: na - task: run_julia_benchmarks status: na - task: run_python_benchmarks status: na - task: run_r_benchmarks status: na - task: run_javascript_tests status: na ---

anandkaranubc · 2025-01-26T18:49:41Z

The README link for the weighted standard deviation needs to be updated with a more appropriate reference. Currently, there does not seem to be a specific Wikipedia link for the weighted standard deviation.

Current README link:

stdlib/lib/node_modules/@stdlib/stats/incr/wstdev/README.md

Lines 151 to 153 in 9a3a0e7

    
           <!-- NOTE: The link below needs to be updated to a more accurate or relevant source. --> 
        
           [weighted-standard-deviation]: https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_variance

Additionally, considering the reference implementation stats/incr/wmean, it assumes that weights are always positive. However, this assumption is not explicitly documented anywhere. Should this be clarified? If necessary, I can update the documentation and implementations for both wmean and wstdev to explicitly mention that weights must be positive. Or, the implementation can be modified to include guards before using wsum.

Code snippet:

stdlib/lib/node_modules/@stdlib/stats/incr/wmean/lib/main.js

Line 112 in b0b66d2

mu += ( w/wsum ) * ( x-mu );

As seen above, if wsum is 0, then mu is undefined.

kgryte · 2025-01-26T22:34:16Z

Currently, there does not seem to be a specific Wikipedia link for the weighted standard deviation.

Given the close relationship between the variance and standard deviation, there won't be a specific link for the standard deviation. The closest reference will be the variance. There aren't any standard deviation-specific algorithms, which don't arise first from computing the variance.

it assumes that weights are always positive.

You're right. In wmean, we should allow for nonnegative weights (i.e., w_i = 0) and only start updating mu once we encounter a non-zero weight.

--- type: pre_commit_static_analysis_report description: Results of running static analysis checks when committing changes. report: - task: lint_filenames status: passed - task: lint_editorconfig status: passed - task: lint_markdown status: passed - task: lint_package_json status: na - task: lint_repl_help status: passed - task: lint_javascript_src status: passed - task: lint_javascript_cli status: na - task: lint_javascript_examples status: na - task: lint_javascript_tests status: passed - task: lint_javascript_benchmarks status: na - task: lint_python status: na - task: lint_r status: na - task: lint_c_src status: na - task: lint_c_examples status: na - task: lint_c_benchmarks status: na - task: lint_c_tests_fixtures status: na - task: lint_shell status: na - task: lint_typescript_declarations status: passed - task: lint_typescript_tests status: na - task: lint_license_headers status: passed ---

--- type: pre_commit_static_analysis_report description: Results of running static analysis checks when committing changes. report: - task: lint_filenames status: passed - task: lint_editorconfig status: passed - task: lint_markdown status: na - task: lint_package_json status: na - task: lint_repl_help status: na - task: lint_javascript_src status: passed - task: lint_javascript_cli status: na - task: lint_javascript_examples status: na - task: lint_javascript_tests status: passed - task: lint_javascript_benchmarks status: na - task: lint_python status: na - task: lint_r status: na - task: lint_c_src status: na - task: lint_c_examples status: na - task: lint_c_benchmarks status: na - task: lint_c_tests_fixtures status: na - task: lint_shell status: na - task: lint_typescript_declarations status: na - task: lint_typescript_tests status: na - task: lint_license_headers status: passed --- --- type: pre_push_report description: Results of running various checks prior to pushing changes. report: - task: run_javascript_examples status: na - task: run_c_examples status: na - task: run_cpp_examples status: na - task: run_javascript_readme_examples status: passed - task: run_c_benchmarks status: na - task: run_cpp_benchmarks status: na - task: run_fortran_benchmarks status: na - task: run_javascript_benchmarks status: na - task: run_julia_benchmarks status: na - task: run_python_benchmarks status: na - task: run_r_benchmarks status: na - task: run_javascript_tests status: passed ---

…andard deviation' --- type: pre_commit_static_analysis_report description: Results of running static analysis checks when committing changes. report: - task: lint_filenames status: passed - task: lint_editorconfig status: passed - task: lint_markdown status: passed - task: lint_package_json status: passed - task: lint_repl_help status: na - task: lint_javascript_src status: passed - task: lint_javascript_cli status: na - task: lint_javascript_examples status: passed - task: lint_javascript_tests status: na - task: lint_javascript_benchmarks status: na - task: lint_python status: na - task: lint_r status: na - task: lint_c_src status: na - task: lint_c_examples status: na - task: lint_c_benchmarks status: na - task: lint_c_tests_fixtures status: na - task: lint_shell status: na - task: lint_typescript_declarations status: na - task: lint_typescript_tests status: na - task: lint_license_headers status: passed --- --- type: pre_push_report description: Results of running various checks prior to pushing changes. report: - task: run_javascript_examples status: passed - task: run_c_examples status: na - task: run_cpp_examples status: na - task: run_javascript_readme_examples status: passed - task: run_c_benchmarks status: na - task: run_cpp_benchmarks status: na - task: run_fortran_benchmarks status: na - task: run_javascript_benchmarks status: na - task: run_julia_benchmarks status: na - task: run_python_benchmarks status: na - task: run_r_benchmarks status: na - task: run_javascript_tests status: na ---

anandkaranubc · 2025-01-28T23:03:58Z

@kgryte Open to your feedback on this complete PR.

SVG has been updated. Thanks for the help in OH.

This is the part where I would need your attention the most.

stdlib/lib/node_modules/@stdlib/stats/incr/wstdev/lib/main.js

Line 34 in c790ae8

* ## Method

Also, I changed every occurrence of "weighted standard deviation" to "weighted sample standard deviation" to be consistent with the other standard deviation related implementations in stats/incr/*. I hope this change was essential.

--- type: pre_commit_static_analysis_report description: Results of running static analysis checks when committing changes. report: - task: lint_filenames status: passed - task: lint_editorconfig status: passed - task: lint_markdown status: passed - task: lint_package_json status: na - task: lint_repl_help status: na - task: lint_javascript_src status: na - task: lint_javascript_cli status: na - task: lint_javascript_examples status: na - task: lint_javascript_tests status: na - task: lint_javascript_benchmarks status: na - task: lint_python status: na - task: lint_r status: na - task: lint_c_src status: na - task: lint_c_examples status: na - task: lint_c_benchmarks status: na - task: lint_c_tests_fixtures status: na - task: lint_shell status: na - task: lint_typescript_declarations status: na - task: lint_typescript_tests status: na - task: lint_license_headers status: passed ---

--- type: pre_commit_static_analysis_report description: Results of running static analysis checks when committing changes. report: - task: lint_filenames status: passed - task: lint_editorconfig status: passed - task: lint_markdown status: passed - task: lint_package_json status: na - task: lint_repl_help status: na - task: lint_javascript_src status: na - task: lint_javascript_cli status: na - task: lint_javascript_examples status: na - task: lint_javascript_tests status: na - task: lint_javascript_benchmarks status: na - task: lint_python status: na - task: lint_r status: na - task: lint_c_src status: na - task: lint_c_examples status: na - task: lint_c_benchmarks status: na - task: lint_c_tests_fixtures status: na - task: lint_shell status: na - task: lint_typescript_declarations status: na - task: lint_typescript_tests status: na - task: lint_license_headers status: passed --- --- type: pre_push_report description: Results of running various checks prior to pushing changes. report: - task: run_javascript_examples status: na - task: run_c_examples status: na - task: run_cpp_examples status: na - task: run_javascript_readme_examples status: passed - task: run_c_benchmarks status: na - task: run_cpp_benchmarks status: na - task: run_fortran_benchmarks status: na - task: run_javascript_benchmarks status: na - task: run_julia_benchmarks status: na - task: run_python_benchmarks status: na - task: run_r_benchmarks status: na - task: run_javascript_tests status: na ---

--- type: pre_commit_static_analysis_report description: Results of running static analysis checks when committing changes. report: - task: lint_filenames status: passed - task: lint_editorconfig status: passed - task: lint_markdown status: na - task: lint_package_json status: na - task: lint_repl_help status: na - task: lint_javascript_src status: na - task: lint_javascript_cli status: na - task: lint_javascript_examples status: na - task: lint_javascript_tests status: na - task: lint_javascript_benchmarks status: na - task: lint_python status: na - task: lint_r status: na - task: lint_c_src status: na - task: lint_c_examples status: na - task: lint_c_benchmarks status: na - task: lint_c_tests_fixtures status: na - task: lint_shell status: na - task: lint_typescript_declarations status: na - task: lint_typescript_tests status: na - task: lint_license_headers status: passed --- --- type: pre_push_report description: Results of running various checks prior to pushing changes. report: - task: run_javascript_examples status: na - task: run_c_examples status: na - task: run_cpp_examples status: na - task: run_javascript_readme_examples status: na - task: run_c_benchmarks status: na - task: run_cpp_benchmarks status: na - task: run_fortran_benchmarks status: na - task: run_javascript_benchmarks status: na - task: run_julia_benchmarks status: na - task: run_python_benchmarks status: na - task: run_r_benchmarks status: na - task: run_javascript_tests status: na ---

--- type: pre_commit_static_analysis_report description: Results of running static analysis checks when committing changes. report: - task: lint_filenames status: passed - task: lint_editorconfig status: passed - task: lint_markdown status: passed - task: lint_package_json status: na - task: lint_repl_help status: passed - task: lint_javascript_src status: passed - task: lint_javascript_cli status: na - task: lint_javascript_examples status: passed - task: lint_javascript_tests status: na - task: lint_javascript_benchmarks status: na - task: lint_python status: na - task: lint_r status: na - task: lint_c_src status: na - task: lint_c_examples status: na - task: lint_c_benchmarks status: na - task: lint_c_tests_fixtures status: na - task: lint_shell status: na - task: lint_typescript_declarations status: passed - task: lint_typescript_tests status: na - task: lint_license_headers status: passed --- --- type: pre_push_report description: Results of running various checks prior to pushing changes. report: - task: run_javascript_examples status: passed - task: run_c_examples status: na - task: run_cpp_examples status: na - task: run_javascript_readme_examples status: passed - task: run_c_benchmarks status: na - task: run_cpp_benchmarks status: na - task: run_fortran_benchmarks status: na - task: run_javascript_benchmarks status: na - task: run_julia_benchmarks status: na - task: run_python_benchmarks status: na - task: run_r_benchmarks status: na - task: run_javascript_tests status: na ---

anandkaranubc · 2025-02-17T02:37:28Z

/stdlib merge

stdlib-bot · 2025-02-17T02:40:53Z

Coverage Report

Package	Statements	Branches	Functions	Lines
stats/incr/wstdev	$\color{red}254/260$ $\color{green}+97.69\%$	$\color{red}22/25$ $\color{green}+88.00\%$	$\color{green}3/3$ $\color{green}+100.00\%$	$\color{red}254/260$ $\color{green}+97.69\%$

The above coverage report was generated for the changes in this PR.

kgryte · 2025-04-30T01:36:02Z

/stdlib merge

kgryte · 2025-04-30T01:36:30Z

lib/node_modules/@stdlib/stats/incr/wstdev/README.md

+
+<section class="related">
+
+* * *


This is a new package, so all these related links should be removed.

kgryte · 2025-04-30T01:37:35Z

lib/node_modules/@stdlib/stats/incr/wstdev/README.md

+var accumulator;
+var v;
+var w;
+var i;
+
+// Initialize an accumulator:
+accumulator = incrwstdev();
+
+// For each simulated datum, update the weighted sample standard deviation...
+for ( i = 0; i < 100; i++ ) {
+    v = uniform( 0.0, 100.0 );
+    w = uniform( 0.0, 100.0 );
+    accumulator( v, w );
+}
+console.log( accumulator() );


Suggested change

var accumulator;

var v;

var w;

var i;

// Initialize an accumulator:

accumulator = incrwstdev();

// For each simulated datum, update the weighted sample standard deviation...

for ( i = 0; i < 100; i++ ) {

v = uniform( 0.0, 100.0 );

w = uniform( 0.0, 100.0 );

accumulator( v, w );

}

console.log( accumulator() );

// Initialize an accumulator:

var accumulator = incrwstdev();

// For each simulated datum, update the weighted sample standard deviation...

var i;

for ( i = 0; i < 100; i++ ) {

accumulator( uniform( 0.0, 100.0 ), uniform( 0.0, 100.0 ) );

}

console.log( accumulator() );

To avoid additional variables. Bigger picture, we want to make ES5 examples as easy to translate to ES6+ as possible in an automated fashion.

kgryte · 2025-04-30T01:40:13Z

lib/node_modules/@stdlib/stats/incr/wstdev/docs/repl.txt

@@ -0,0 +1,44 @@
+
+{{alias}}( [mean] )


Looks like we have documentation inconsistency here. The README does not indicate that a known mean is supported.

kgryte · 2025-04-30T01:41:53Z

lib/node_modules/@stdlib/stats/incr/wstdev/examples/index.js

+var accumulator;
+var s;
+var x;
+var w;
+var i;
+
+// Initialize an accumulator:
+accumulator = incrwstdev();
+
+// For each simulated datum, update the weighted sample standard deviation...
+console.log( '\nValue\tWeight\tSigma\n' );


Suggested change

var accumulator;

var s;

var x;

var w;

var i;

// Initialize an accumulator:

accumulator = incrwstdev();

// For each simulated datum, update the weighted sample standard deviation...

console.log( '\nValue\tWeight\tSigma\n' );

// Initialize an accumulator:

var accumulator = incrwstdev();

// For each simulated datum, update the weighted sample standard deviation...

console.log( '\nValue\tWeight\tSigma\n' );

var s;

var x;

var w;

var i;

Ideally, variables are declared in examples as closely as possible to where they are first assigned, but not within blocks.

kgryte · 2025-04-30T01:42:35Z

lib/node_modules/@stdlib/stats/incr/wstdev/lib/main.js

+*     ```
+*
+* @param {number} [mean] - mean value
+* @throws {TypeError} must provide a number primitive


Suggested change

* @throws {TypeError} must provide a number primitive

* @throws {TypeError} must provide a number

kgryte · 2025-04-30T01:43:18Z

lib/node_modules/@stdlib/stats/incr/wstdev/docs/types/index.d.ts

+type accumulator = ( x?: number, w?: number ) => number | null;
+
+/**
+* Returns an accumulator function which incrementally computes a weighted sample standard deviation.


Looks like this is correct, based on the implementation, where a known mean parameter is supported. This and the tests need to be updated.

kgryte · 2025-04-30T01:44:01Z

lib/node_modules/@stdlib/stats/incr/wstdev/lib/main.js

+* // returns ~0.478
+*
+* @example
+* var accumulator = incrwstdev( 3.0 );


Is there more to this example? Do we want to show computed values, as done in the prior example?

kgryte · 2025-04-30T01:49:19Z

lib/node_modules/@stdlib/stats/incr/wstdev/test/test.js

+		xwSum += x * w;
+		wSum += w;
+		wMean = xwSum / wSum;
+		wVarSum += w * (x - wMeanPrev) * (x - wMean);
+		wMeanPrev = wMean;
+		expected = sqrt( wVarSum / wSum );


Instead of doing it this way, I suggest computing all the expected values manually (e.g., by hand). While this looks correct, prefer computing using, e.g., textbook algorithms from first principles, rather than reimplement another incremental algorithm.

kgryte

Left an initial round of comments.

anandkaranubc added 6 commits January 26, 2025 10:13

anandkaranubc added 3 commits January 28, 2025 13:33

anandkaranubc added 4 commits January 28, 2025 15:14

stdlib-bot added bot: In Progress Pull request is currently awaiting automation. Statistics Issue or pull request related to statistical functionality. labels Feb 17, 2025

Merge remote-tracking branch 'upstream/develop' into incr-wstdev

e97076d

stdlib-bot removed the bot: In Progress Pull request is currently awaiting automation. label Feb 17, 2025

Planeshifter added the Potential Duplicate There might be another pull request resolving the same issue. label Mar 15, 2025

stdlib-bot added the bot: In Progress Pull request is currently awaiting automation. label Apr 30, 2025

kgryte reviewed Apr 30, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/develop' into incr-wstdev

5872553

stdlib-bot removed the bot: In Progress Pull request is currently awaiting automation. label Apr 30, 2025

kgryte reviewed Apr 30, 2025

View reviewed changes

kgryte requested changes Apr 30, 2025

View reviewed changes

stdlib-bot removed the Potential Duplicate There might be another pull request resolving the same issue. label Apr 30, 2025

anandkaranubc marked this pull request as draft April 30, 2025 15:01

Merge branch 'develop' into incr-wstdev

04b6d65

	* @throws {TypeError} must provide a number primitive
	* @throws {TypeError} must provide a number


		<section class="related">

		* * *

Uh oh!

feat: add stats/incr/wstdev #4914

Are you sure you want to change the base?

feat: add stats/incr/wstdev #4914

Uh oh!

Conversation

anandkaranubc commented Jan 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Questions

Other

Checklist

Uh oh!

anandkaranubc commented Jan 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kgryte commented Jan 26, 2025

Uh oh!

anandkaranubc commented Jan 28, 2025

Uh oh!

anandkaranubc commented Feb 17, 2025

Uh oh!

stdlib-bot commented Feb 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report

Uh oh!

kgryte commented Apr 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kgryte left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat: add `stats/incr/wstdev` #4914

feat: add `stats/incr/wstdev` #4914

anandkaranubc commented Jan 26, 2025 •

edited

Loading

anandkaranubc commented Jan 26, 2025 •

edited

Loading

stdlib-bot commented Feb 17, 2025 •

edited

Loading